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ABSTRACT 



Pursuant to a congressional request, the United States 
General Accounting Office (GAO) has provided information on math and science 
education that focuses on: (1) key efforts undertaken by the Department of 

Education, the National Institutes of Health (NIH) , and the nine federal 
agencies under the purview of the House Committee on Science that supported 
the development and dissemination of comprehensive kindergarten through 
12th-grade mathematics and science curriculum materials and the 
characteristics of these efforts; (2) the extent to which federal agencies 
coordinated their efforts to develop and disseminate these materials; and (3) 
the extent to which agencies assessed the quality and use of these materials. 
Data collection methods consisted of interviewing agency officials, reviewing 
agency documents, and surveying agencies. Some results indicate that GAO 
agencies used a variety of approaches to develop, promote awareness of, and 
distribute mathematics and science materials for K-12; of the 11 agencies GAO 
surveyed, nine identified 210 projects that were actively developing 
mathematics and science materials in fiscal year 1999, and the Department of 
Education and the National Science Foundation (NSF) supported the development 
of two-thirds of these projects as part of their efforts to improve the 
teaching and learning of math and science. (ASK) 
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Human Services Division 
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July 12, 2000 
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Ranking Minority Member 
Committee on Science 
House of Representatives 

The Honorable Eddie Bernice Johnson 
Ranking Minority Member 
Subcommittee on Basic Research 
Committee on Science 
House of Representatives 

In fiscal year 1999, the federal government invested an estimated $2.5 
billion in mathematics and science education. This investment, which was 
spread across multiple federal agencies, supported a wide range of 
activities — from research on and development of curriculum materials to 
teacher training and professional development. However, recent 
assessments of student performance in mathematics and science 
demonstrated that, although U.S. fourth graders did relatively well in both 
mathematics and science, 12th-grade students were far from the goals set 
by the Bush administration and the 50 state governors of being first in the 
world by the year 2000. As a result, there is a growing concern that United 
States mathematics and science education may not be providing students 
with the skills necessary to succeed in the work place or in a global 
economy. 
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Results in Brief 



Given these concerns, you asked us to determine (1) the key efforts 
undertaken by the Department of Education (Education), the National 
Institutes of Health (NIH), and the nine federal agencies under the purview 
of the House Committee on Science 1 that supported the development and 
dissemination of comprehensive kindergarten through 12th-grade 
mathematics and science curriculum materials 2 and the characteristics of 
these efforts; (2) the extent to which federal agencies coordinated their 
efforts to develop and disseminate these materials; and (3) the extent to 
which agencies assessed the quality and use of these materials. To obtain 
this information we interviewed agency officials and members of 
professional associations and reviewed agency documents. We also 
surveyed the 1 1 agencies to determine how they generally developed, 
disseminated, and evaluated these types of materials. Within these 
agencies, we conducted a second survey of curriculum development 
projects that were active in fiscal year 1999. 3 We performed our work 
between June 1999 and June 2000 in accordance with generally accepted 
government auditing standards. 



Agencies used a variety of approaches to develop, promote awareness of, 
and distribute mathematics and science materials for kindergarten through 
12th grade. Of the 11 agencies we surveyed, nine identified 210 projects 
that were actively developing mathematics and science materials in fiscal 
year 1999. Education and NSF supported the development of two-thirds of 
these projects as part of their effort to improve the teaching and learning of 
math and science. However, most other agencies we reviewed generally 
developed these materials as part of their goal to increase public awareness 



'The Department of Energy (DOE), the Environmental Protection Agency (EPA), the 
Federal Aviation Administration (FAA), the Federal Emergency Management Agency 
(FEMA), the National Aeronautics and Space Administration (NASA), the National Institute 
of Standards and Technology (NIST), the National Oceanic and Atmospheric Administration 
(NOAA), the National Science Foundation (NSF), and the U.S. Geological Survey (USGS). 

Materials refer to comprehensive materials sets that contain all of the components a 
teacher would need to teach and a student would need to learn one or more mathematics 
and/or science concepts. In some cases, agencies developed materials directly using agency 
staff. In other cases, agencies provided funding or expertise to others. 

3 FEMA and NIST did not develop materials in fiscal year 1999. NSF materials under 
development in fiscal year 1999 did not become available for use during fiscal year 1999. 
NSF materials developed with funds from previous years did become available during fiscal 
year 1999, but are not included in this study because they did not fall within the criteria 
established for inclusion. 
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of a specific subject, rather than to improve math and science education. 
Overall, the materials incorporated the use of technology, such as 
computer software and the Internet, and projects tended to focus on 
science rather than math. To promote and distribute these materials, 
agencies generally relied most heavily on resources that were available 
within the agency, such as agency-sponsored conferences and workshops 
and agency publication distribution centers. In addition, agency officials 
reported that there is a trend toward making the materials available on-line 
through the Internet. 

To some extent, agencies producing math and science materials 
coordinated their efforts within and across agencies and with state and 
local organizations. Of the nine agencies that developed math and science 
materials in fiscal year 1999, four had some interned mechanism for 
coordinating some or all of their agency’s projects. Some agencies also 
coordinated efforts on an informal basis. Education and NSF, two agencies 
that share the goals of improving teaching and learning of math and 
science, worked with other agencies through an informed network of 
individuals involved in the development of materials. NIH coordinated its 
efforts through intraagency communication channels such as the Science 
Education Resource Group and the Office of Science Education. Two other 
agencies with related goals, EPA and USGS, also coordinated efforts with 
each other as part of their efforts to promote public awareness of scientific 
information. Some agencies also made some effort to coordinate with state 
and local organizations in developing math and science education 
materials. Agencies generally relied on their own distribution mechanisms 
to disseminate information about materials rather than coordinating 
dissemination efforts with other agencies. Although the National 
Clearinghouse for Mathematics and Science Education (Eisenhower 
Clearinghouse) was established to serve as a central source of information 
about math and science materials, most agencies did not send the 
clearinghouse copies of all their math and science materials. As a result, no 
single comprehensive resource for all federally sponsored materials is 
available to educators seeking to make informed decisions about these 
materials. 
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Information was available about the quality of some of the federally 
developed mathematics and science materials, but little information was 
available on the extent to which these materials were used in the 
classroom. Agencies used a range of methodologies to assess the quality of 
materials, including piloting and field testing of the materials and the 
review of customer feedback obtained through comment cards and 
surveys. Of the 61 materials that became available in fiscal year 1999, about 
half had been assessed for quality at the time of our review. 

We make recommendations to ensure that agencies submit their math and 
science educational materials to the Eisenhower Clearinghouse and 
evaluate the quality of the educational materials they develop. The agencies 
that commented on the report generally supported our recommendations. 


Background 


Education is primarily a state and local function; however, the federal 
government plays an important role as a source of information, technical 
and financial assistance, and leadership. Several laws, including the 
Elementary and Secondary Education Act of 1965 (ESEA), authorize 
programs that encourage research on and development of innovative 
teaching strategies and curriculum materials that are tied to curriculum 
content standards, as well as expanded technical assistance to implement 
these innovations. Federal agencies generally produce these materials as 
part of their educational research and development efforts. It is the state 
and local school districts, however, that decide whether or not they will be 
adopted. 

On a national level, federal education programs such as Title I of the ESEA 
and Goals 2000 have stressed accountability for student achievement and 
the development of curriculum content standards and related assessments. 
Several national organizations, including the National Council of Teachers 
of Mathematics, the National Science Teachers Association, the National 
Research Council of the National Academy of Sciences, and the American 
Association for the Advancement of Science, have developed curriculum 
standards for mathematics and science education. These standards were 
designed to establish a broad framework of what mathematics and science 
curricula should include, such as the topics that should be taught and the 
extent to which they should be emphasized. States, local school districts, 
and those who develop educational materials have wide discretion in 
whether and how they adopt such standards. Similarly, curriculum 
materials may be selected in a number of ways, depending on the priorities 
of schools and districts. States and local school districts establish 
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frameworks that specify the knowledge and skills that students should 
have at particular points in time and guide the selection of instructional 
strategies and materials to accomplish these ends. Not all districts and 
schools, however, use these frameworks to adopt materials in the same 
way. For example, some districts adopt a single set of materials for use in 
all classrooms, while other districts allow individual schools to select their 
own materials from an approved list. Other districts and schools 
supplement purchased materials with modules that are particularly 
relevant to local issues. 

To make math and science materials available to educators, the Excellence 
in Mathematics, Science and Engineering Act of 1990 authorized the 
establishment of the clearinghouse now known as the Eisenhower 
Clearinghouse. The Eisenhower Clearinghouse serves as a permanent 
repository of math and science education instructional and curriculum 
materials for kindergarten through 12th-grade educators and disseminates 
information about these materials to the public. The ESEA also requires 
each federal agency or department that develops mathematics or science 
instructional material or programs to submit copies of the material or 
program to the Eisenhower Clearinghouse. 



Agencies Used a 
Variety of Methods to 
Develop and 
Disseminate Materials 



Agencies used a variety of approaches to develop, promote, and distribute 
math and science materials for kindergarten through 12th grade. Education 
and NSF funded two-thirds of the 210 projects that produced math and 
science materials in fiscal year 1999. Of the nine agencies, NSF, Education, 
and NIH supported the development of educational materials as part of 
their efforts to improve math and science teaching and learning. The other 
agencies we surveyed developed materials as part of their goal to increase 
public awareness about math and science topics related to their agency’s 
missions. To promote and distribute these materials, agencies generally 
relied most heavily on resources available within the agency, although 
agency officials reported an increasing trend toward distributing materials 
through the Internet. 
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NSF and Education Provide 
Most Support for the 
Development of Math and 
Science Materials 



The agencies we surveyed varied not only in terms of the number of 
projects they were developing, but also in terms of the cost of these 
projects. Of the 210 projects we identified, NSF and Education funded 143, 
or two-thirds. 4 The nine agencies that had math and science projects active 
in fiscal year 1999 estimated that these projects would cost $314 million. 5 
About one-fourth of the projects had an expected final cost of less than 
$100,000 while about one-third had an expected cost of $1 million or more. 
NSF and Education supported the development of almost 90 percent of the 
projects costing $1 million or more. Projects developed by DOE, EPA, FAA, 
and USGS cost less than $300,000 each. Table 1 shows the number and cost 
of projects by agencies. 



Table 1 : Math and Science Projects Active in Fiscal Year 1999 



Agency * 


Number of projects 


Reported estimated total project cost 
(in thousands of dollars) 8 


NSF 


93 


$106,000 


Education 


50 


179,000 


NIH 


18 


11,130 


NOAA 


15 


12,760 b 


EPA 


13 


800 


USGS 


10 


288 


NASA 


4 


3,000 


DOE 


5 


880 


FAA 


2 


188 c 


Total 


210 


$314,046 



a This includes the reported estimated cost for all years of multiyear products. 

‘This includes the estimated total cost of the GLOBE program, which is a multiagency project 
combining NOAA, NASA, and NSF. 

The funds for these projects came from sources other than FAA. 



Education generally supports the development of curriculum materials rather than 
developing them itself. Education’s development efforts are generally limited to materials 
for special populations. 

5 This includes the reported estimated cost for all years of multiyear projects. 
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Each agency produced materials that reflected its missions. Although all 
the agencies had missions related to math and science, three agencies — 
Education, NSF, and NIH — had missions that focus specifically on 
improving math and science teaching and learning. NSF’s mission, for 
example, is to “initiate and support science and engineering education 
programs.” Consequently, before NSF funded a project, it expected 
grantees to show that the project would produce materials that furthered 
educational practices. In contrast, other agencies developed materials that 
reflected their scientific missions. In doing so, these materials incorporated 
math or science; however, improving math and science instruction was not 
their primary purpose. For example, one of NOAA’s missions is to manage 
and develop coastal ecosystems. To educate students about water 
resources, NOAA worked with other federal agencies to develop materials 
on hazards to coastal and ocean areas. Although agencies developed 
materials for all grade levels, the larger number of projects developed 
materials for higher versus lower grades, as shown in fig. 1. 
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Figure 1: Target Audience of Materials in Development During Fiscal Year 1999 
Number of Materials 




Kindergarten 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 



Most of the projects focused on science or a combination of math and 
science and relatively few focused solely on math; NSF and Education 
funded the projects that included a math focus. Fig. 2 shows the proportion 
of projects that focused on math, science, or a combination of the two. 
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Figure 2: Primary Orientation of Projects in Development During Fiscal Year 1999 



Primarily Oriented 
Toward Science 

126 




Oriented Toward 
Both Math and Science 

48 



Primarily Oriented 
Toward Math 
36 



The 210 projects made 61 materials available in fiscal year 1999. These 
incorporated technology and emphasized formed instruction. More than 80 
percent of these materials provided an opportunity for students and 
teachers to use computer software or Internet sites, even when in some 
cases this technology was not necessary to use the materials. Although 
some of the materials were designed for informal instruction outside the 
classroom, most were intended for formal classroom instruction. About 
half the materials incorporated either state or voluntary national education 
standards. Table 2 provides three examples of some of the types of 
materials agencies created. For more detail on the characteristics of the 61 
materials that became available during fiscal year 1999, see app. n. 
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Table 2: Examples of Materials Produced by Projects 


Agency 


Project 


Materials 


NIH 


Understanding 
Gene Testing 


Lecture with slides; booklet on 
frequently asked questions about 
gene testing 


FAA 


Aviation in a Bag 


Lesson plans on aviation- related 
science principles, with worksheets 


Council on Environmental 
Quality, Education, EPA, 
NASA, NOAA, NSF, Office 
of Science Technology 
Policy, Peace Corps, Dept, 
of State, AID, a USGS 


The GLOBE 
Program 


Web-based interactive program with 
daily, weekly, and seasonal lesson 
plans through which students learn 
scientific research techniques as 
they study the environment 



“Agency for International Development. 



Agencies Used Multiple 
Approaches to Promote and 
Distribute Materials 



In promoting materials, agencies relied most heavily on resources that 
were available within the agency, such as agency-sponsored conferences 
and workshops, the agency website, or the agency publication distribution 
centers. Workshops and conferences sponsored by nonfederal 
organizations or state and local agencies also provided a popular means of 
promoting materials. As part of their efforts to promote materials, agency 
officials also reported targeting professional organizations and 
clearinghouses devoted to specific scientific subjects or topics. For 
example, the USGS reported targeting its materials to the National Council 
of Geography Educators, the American Geographical Society, and the 
National Science Teacher’s Association. Officials in other agencies reported 
using mailing lists from professional organizations in order to target 
teachers with interests in specific subjects. Fig. 3 shows the different 
methods that agencies used to promote materials. 
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Figure 3: Approaches Used by Agencies to Promote Awareness of Materials Produced in Fiscal Year 1999 



Approaches 



Agency Conferences/ 
Workshops 




Agency's 
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Nonfederal 
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Percentage 



100 



Note: NSF materials are not included in this chart because NSF did not identify any materials under 
development with fiscal year 1999 funds that became available in fiscal year 1999. According to 
agency officials, NSF requires that its products be submitted to the Eisenhower Clearinghouse. Most 
NSF products are distributed by publishers. 



Some approaches to distributing materials were specifically targeted to 
educators, while other approaches were more general. Many of the 
methods used to promote awareness of materials were also used as 
methods of distribution. For example, agencies used workshops not only to 
make educators aware of materials, but also to distribute materials. One 
agency, NASA, distributed materials through resource centers specifically 
for educators located in every state. These are walk-in centers where 
teachers can browse through materials developed by NASA as well as 
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materials on related subjects developed commercially or by other agencies 
and organizations. Educators can also call the center and get assistance 
choosing materials, learn about professional development opportunities, 
and have materials and information mailed to them. In contrast, some 
agencies made no distinction between their general audience and 
educators when distributing materials. These agencies distributed math 
and science materials through the warehouses and shipping centers used 
for all agency publications. For example, several divisions within EPA used 
the National Service Center for Environmental Publications and 
Information to distribute their materials. The center published a catalog of 
EPA publications, but did not separately categorize educational materials. 
Educators could order materials from the catalog just as they would order 
any other EPA publication. USGS used a combination of approaches. It had 
a warehouse and shipping center, but also operated eight Earth Science 
Information Centers. These centers provided information and sales service 
for all USGS map products and earth science publications but were not 
targeted to educators. Finally, other agencies relied on external 
distributors. For example, NSF required grantees to secure a private 
publisher as part of their grant. These grantees were then responsible for 
disseminating the materials. 

Although agencies generally relied on the conferences and workshops they 
sponsored and their publication distribution centers to distribute materials, 
officials reported a trend toward putting materials online. Many agencies 
found that putting materials in formats that could be accessed online was 
more cost-effective than incurring publishing and shipping costs. However, 
not all schools and teachers have access to the Internet. Even in schools 
with access, teachers may have limited time on the computers. 
Furthermore, some officials told us that the schools’ equipment might not 
be adequate for downloading and printing materials. Some of the materials 
agencies developed were colorful and do not print well in black and white. 
Other materials, such as posters or videos, may not lend themselves to 
downloading and printing. 



Agency Coordination 
of Material 
Development and 
Dissemination Uneven 



The 11 agencies we surveyed made some efforts to coordinate the 
development of math and science materials internally, across agencies, and 
with state and local organizations. Four of the agencies had a mechanism 
to coordinate the development of at least some of their materials. Some 
agencies coordinated materials development with each other on some of 
their projects. Agencies also coordinated the development of materials 
with other nonfederal entities. Most agencies relied on their own 
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Coordination Efforts Varied 
Within and Across Agencies 



workshops or distribution centers to disseminate materials rather than 
coordinating with other agencies. The Eisenhower Clearinghouse was 
established to serve as a central source of information about math and 
science materials, and agencies are required by law to send copies of their 
materials to the Eisenhower Clearinghouse; however, only six agencies 
reported sending copies of at least some of the materials they developed in 
fiscal year 1999 to the Eisenhower Clearinghouse. 



Of the 1 1 agencies we surveyed, the nine that were developing math and 
science materials in fiscal year 1999 varied in the extent to which efforts 
within the agency were coordinated. Four agencies had mechanisms to 
coordinate the development of some of the materials produced by that 
agency. NSF was the only agency we surveyed that distributed all of its 
grants for kindergarten through 12th-grade math and science materials 
through one office. All the other agencies that were developing materials 
did so in at least two offices. Some efforts by these separate offices to 
develop materials were coordinated, while others were not. According to 
agency officials, NIH coordinated its materials development efforts 
through intraagency coordination systems such as the Science Education 
Resource Group and the Office of Science Education (OSE), which 
increase the awareness of materials development projects in other NIH 
institutes and centers. In addition, the OSE has consulted with NIH 
institutes on a number of science education products and serves as a 
source of educational advice and expertise. 

NSF and Education, two agencies that share the goal of improving the 
teaching and learning of math and science, coordinated the development of 
materials through an informal network of individuals. NSF officials told us 
that these informal networks allowed them to achieve their common goals 
with a high level of creativity and a minimum amount of duplication. 

Other agencies with related, although somewhat different, goals also 
coordinated efforts with each other. For example, both EPA and USGS had 
goals to promote public awareness of scientific information. EPA’s strategic 
plan included the goal of ensuring that all parts of society have access to 
accurate information sufficient to effectively participate in managing 
human health and environmental risks. USGS’ strategic plan included the 
goal of providing reliable scientific information. USGS and EPA, along with 
several other federal agencies, developed the Water Matters project, which 
covered topics including Oceans and Coastal Hazards and Hazardous 
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Waste and Watersheds. By combining their efforts, EPA and USGS were 
each able to further their goals. 



Limited Compliance With 
Requirement to Submit 
Materials to Eisenhower 
Clearinghouse 



Agencies also coordinated efforts with state and local organizations. Of the 
42 projects that reported developing materials in partnership with others, 
36 indicated coordination with nonfederal partners. Nonfederal partners 
included state education agencies, local school districts, individual 
schools, professional organizations, universities, nonprofit organizations, 
and television stations. For example, Coastal America, a partnership 
between federal agencies with stewardship responsibilities for the nation’s 
coastal habitats and resources, worked with local aquariums and related 
organizations around the nation to promote education on coastal 
ecosystems. The Aurora project, funded by Education, worked with public 
school districts and private schools in Oklahoma along with Southwestern 
Oklahoma State University and the Southwest Educational Development 
Laboratory to create an interactive, online curriculum where students and 
teachers could find a variety of educational activities related to geology. 



Education’s search of the Eisenhower Clearinghouse and the results of our 
survey showed that most agencies did not submit all their materials to the 
clearinghouse as required by the law. According to Education officials, only 
nine of the materials that became available in fiscal year 1999 were actually 
found in the clearinghouse’s database. 6 Two agencies had policies requiring 
submission of materials to the Eisenhower Clearinghouse. NASA had a 
policy to automatically send copies of materials to the clearinghouse and 
all the materials produced in fiscal year 1999 were in the clearinghouse 
database. Similarly, NSF built a requirement into grants that grantees 
submit copies of materials to the clearinghouse. Several other agencies 
indicated that they did not have policies that require submitting materials, 
but some indicated that they submitted materials to the clearinghouse. 
Some agency officials said they wait to be contacted by Eisenhower staff. 
For example, officials at NIH said that Eisenhower staff contacted them 
once a year, generally in the fall. NIH uses this as a trigger to update NIH 
information in the clearinghouse, including removing materials no longer 
current and adding materials that have passed initial quality assessment 
review and are being disseminated by the agency. At that time NIH staff 
look to see what materials have become available and decide about 



6 Although 61 materials became available in fiscal year 1999, we only had enough information 
to determine the availability of 59 of the materials. 
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whether to submit them based on how the materials have been received by 
their intended audience. The results of our survey showed that about 80 
percent of projects that developed materials available in fiscal year 1999 
did not identify the Eisenhower Clearinghouse as an approach for 
disseminating information about their materials. Some other agency 
officials we surveyed indicated they were not aware of the Eisenhower 
Clearinghouse and its services or were unclear about how to submit 
materials. 

Even though agencies are required by law to submit materials to the 
Eisenhower Clearinghouse, the clearinghouse staff did not depend on 
agencies to do so. They indicated that they use a variety of proactive 
methods to identify and acquire appropriate agency materials. According to 
clearinghouse officials, their acquisition staff of two has contacted federal 
agencies annually. Clearinghouse staff told us that when agencies do not 
coordinate internal efforts to produce math and science materials, the 
clearinghouse staff had difficulty identifying all the materials an agency has 
produced. In such cases, clearinghouse staff had to contact different 
branches or regional offices throughout the agency. Clearinghouse staff 
also learned about materials from teachers, grant announcements, Internet- 
based lists, agency catalogs, and agency exhibits at conferences. They 
acknowledged that they have not had the resources to identify all 
materials. 

Although the Eisenhower Clearinghouse was established to collect and 
distribute information about federally funded math and science materials, 
Education also operates three other educational information 
clearinghouses that could contain references to some math and science 
materials — the Educational Resource Information Center (ERIC), the 
Gateway to Educational Materials (GEM), and the Federal Resources for 
Educational Excellence (FREE) website. However, these clearinghouses 
may either have a specific focus or contain a broader range of materials. 
For example, ERIC serves as a source of information on educational 
research, and generally includes few instructional materials in its database. 
The FREE website provides direct links to federal agencies’ websites that 
have materials on all academic subjects available on-line. GEM provides 
teachers with access to collections of Internet-based educational materials 
available on various federal as well as nonfederal Internet sites, including 
state, university, nonprofit, and commercial Internet sites. All four of the 
clearinghouses have as their target audiences educators, researchers, 
parents of students, and the public. Table 3 shows the characteristics of the 
different clearinghouses. 
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Table 3: Education’s Clearinghouses for Educational Information 


Clearinghouse 


Year 

established 


Subjects 


Types of materials 


Eisenhower 


1992 


Math and Science 


Information on instructional 
materials and federal 
education programs 


ERIC 


1966 


Educational research 
on various academic 
subjects 


Information on some 
instructional materials, but 
limited; mostly research on 
educational practices and 
techniques 


FREE 


1997 


All academic subjects 


Federally funded materials 
available online 


GEM a 


1996 


All academic subjects 


Materials available online 
(most are not federally 
funded) 



fl Although 175 organizations participate in GEM, most of the agencies we surveyed whose materials 
became available in fiscal year 1999 did not. 



Each of these four clearinghouses had a database that included information 
on some of the math and science materials identified in our review. 
However, no single database, including the Eisenhower Clearinghouse, 
contained information on more than 12 of the materials that became 
available in fiscal year 1999. Consequently, no single database could 
provide educators with a comprehensive list of materials available — 
information that would enable them to make informed decisions about the 
full range of federally funded science and math materials available for their 
use. 



Limited Information 
Available on Quality 
and Use of Materials 



Although the 1 1 agencies we surveyed had some information about the 
quality of the materials they produced, they had less information about the 
extent to which these materials were used for classroom instruction. To 
assess quality, agencies used a range of methodologies, including piloting 
and field testing materials and reviewing customer feedback obtained 
through comment cards and surveys. At the time of our review, agencies 
had assessed the quality of about half of the 61 materials that became 
available in fiscal year 1999. Although agencies generally tracked the 
number of materials requested and distributed, they did not collect 
information on whether the materials were actually used in the classroom. 
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The extent to which agencies assessed the quality of the materials they 
developed varied widely across the five agencies that conducted quality 
reviews in fiscal year 1999. Of the 61 materials that became available in 
fiscal year 1999, about half had been assessed for quality, according to our 
survey results (see table 4). Officials from NASA reported that they allowed 
trial usage of materials, collected data from participants and project 
managers, and sought third-party evaluation as means to understand the 
strengths and limitations of the materials produced. Officials in three 
agencies reported that they had not done quality assessments for any of the 
materials that became available through their agency during this period. 

NIH, Education, and NSF told us that their agencies planned to conduct 
more extensive quality assessments of their materials in the future. NIH 
officials indicated that it assesses all its materials before they are 
disseminated through pilot studies or field trials. Officials at Education and 
other agencies also told us that, while they had not assessed the quality of 
all of their materials at the time of our review, they generally intended to do 
so in the future. NSF requires all grantees to assess the quality of the 
materials produced under their grants. In addition, when NSF conducts 
outside reviews of its projects, the quality of the materials produced is also 
evaluated. 



Table 4: Quality Assessment Status of 61 Mathematics and Science Materials That 
Became Available in Fiscal Year 1999, by Agency 



Agency 


Number of materials that 
became available in FY 1999 


Number of materials 
assessed for quality 


Education 


23 


11 


DOE 


1 


0 


EPA 


2 


0 


FAA 


2 


0 


NIH 


9 


4 a 


NOAA 


13 


7 


USGS 


7 


2 


NASA 


4 


4 


Total 


61 


28 


'"Comprehensive evaluation planned but not yet conducted for the five other materials. 



Two organizations have programs to assess the quality of educational 
materials and could serve as means of evaluating the quality of some of the 
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materials produced by the 11 agencies we surveyed. Education and the 
American Association for the Advancement of Science (AAAS) evaluate 
samples of federally and commercially developed mathematics and science 
materials. Education formed a panel of mathematics and science experts 
and practitioners to evaluate educational programs and recommend those 
programs that should be designated as exemplary or promising. AAAS 
Project 2061 has used teams of experienced practicing classroom teachers 
and higher education faculty to identify the extent to which textbooks 
identify key mathematics and science concepts and provide instructional 
strategies likely to help students to learn them. While we do not know if 
Education’s panel of experts or AAAS reviewed any of the 61 materials 
produced in fiscal year 1999, both organizations could be used to evaluate 
materials’ quality, or provide a model for doing so. 

The extent to which educators use agency-developed math and science, 
materials can indicate how well materials have been promoted and their 
perceived quality. Agency officials reported that although they generally 
tracked the number of requests that they received for materials, as well as 
the number of materials they distributed, they generally did not collect 
information on the extent to which the materials were actually used in the 
schools. However, while NASA officials told us that it was difficult to 
obtain information on materials use in the classroom, the feedback that 
they received from users indicated that the materials have been used and 
have been effective. 

Agency officials identified three factors that contributed to the lack of 
information on materials usage. First, if the primary purpose of the 
materials was not to educate kindergarten through 12th-grade students in 
the classroom but rather to increase public awareness of a specific subject 
matter, the materials may have been disseminated through channels for 
noneducational material, making educational materials difficult to identify 
and track. Second, if the materials were recently released for use in the 
classroom, they may be too new to have been widely implemented, 
resulting in limited data on their use beyond that which was available 
during the pilot or field testing of the materials. Third, the wide range of 
approaches that administrators and teachers used to implement the 
materials in the field can significantly affect the extent to which the 
materials are actually used. For example, a June 1999 report on the 
evaluation of NSF’s Instructional Materials Development Program stated 
that teachers modified the materials when they felt uncomfortable with 
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